Skip to content

feat: native engine by default with Python fallback (4.10.0) - #185

Merged
sidmohan0 merged 9 commits into
mainfrom
feature/4.10-convergence
Nov 14, 2026
Merged

sidmohan0 merged 9 commits into
mainfrom
feature/4.10-convergence

Conversation

@sidmohan0

Copy link
Copy Markdown
Contributor

Why

Core is the engine; pip install datafog is the front door. This release connects them. It is a MINOR because nothing is removed — backend="python" restores 4.9.0 behaviour exactly.

Depends on DataFog/datafog-core#233 (Core 1.2.0): the embedded model and the musl/arm64/s390x/ppc64le wheels. Core 1.2.0 must publish first.

What

  • backend defaults to auto: native when datafog-core imports, Python otherwise with one CoreUnavailableWarning. DATAFOG_BACKEND sets the default, backend="python"|"rust" overrides per call. Non-regex engines stay on the Python ML path.
  • Prose PERSON/ADDRESS served by the embedded model over the native path — no spaCy, GLiNER, download or interpreter. PERSON on backend="python" still has no answer, which is now documented instead of implied.
  • Credential/account/UUID types (API_KEY, BEARER_TOKEN, JWT, PRIVATE_KEY, CREDENTIAL_URI, ACCOUNT_NUMBER, US_ROUTING_NUMBER, NPI, UUID) reachable through the existing alias API.
  • datafog.native for the native schema (experimental), datafog.native.version() / .contract_version().
  • VERSIONING.md, docs/retirement.rst, docs/migration-4.10.md, RELEASE_NOTES_4.10.0.md; Development Status: 5 - Production/Stable.
  • Retirement warnings name 4.10.0 as the last supported version with an exact pin, replacing "the final 4.x release".
  • Pin datafog-core>=1.2.0,<2 as a normal dependency.

Reviewed detection differences

Recorded in tests/contracts/rust-1.2.0.json and asserted: Luhn card validation, non-alphanumeric SSN boundaries, NANP area-code 2-9 (the Unix-timestamp guard), strict locale validation. Parity run: 76 match, 3 detector differences, 1 validation difference, 31 outside backend scope.

Verification

  • pytest (non-slow, minus network OCR download): 761 passed, 3 skipped
  • spaCy engine="smart"/"spacy" paths re-run with en_core_web_lg installed: pass
  • OCR and Spark release-gate files still pass or skip on missing Java
  • python -I scripts/check_rust_install.py green against the installed wheel
  • python -m tests.rust_contract matches the reviewed inventory
  • Sphinx build clean
  • Hook cost measured at ~105–110ms/call (was ~78ms); documented honestly, DATAFOG_BACKEND=python recovers it

github-actions Bot and others added 4 commits September 29, 2026 08:02
Resolve backend through 'auto': the native engine when datafog-core
imports, the pure-Python engine otherwise with a single warning.
DATAFOG_BACKEND and backend="python" force the fallback; non-regex
engines stay on the Python ML path.

Serve prose PERSON and ADDRESS from the embedded model in datafog-core,
so names and addresses need no spaCy, GLiNER, download or interpreter.
Expose the new credential, account and UUID entity types through the
alias API and add datafog.native for the native schema.

Deprecation warnings now name 4.10.0 as the last supported version with
an exact pin, and VERSIONING.md plus docs/retirement.rst make the window
and the removal schedule auditable. Nothing is removed in this release.
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ sidmohan0
❌ github-actions[bot]
You have signed the CLA already but the status is still pending? Let us recheck it.

1.2.0 shipped without the exchange-code fix, so 4.10.0 depends on the
release that has it. Contract file follows the engine version.
The corpus encodes Python-engine expectations, so it pins
backend=python by default and scores the native engine under
DATAFOG_ACCURACY_BACKEND=rust against tests/contracts/native-corpus.json,
which records the twelve known divergences with reasons. CI runs both
variants; pre-commit gains a ruff per-file-ignores entry for the dynamic
re-exports in datafog/native.py.
@sidmohan0
sidmohan0 merged commit 8567bd1 into main Nov 14, 2026
63 of 65 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants